Stochastic Optimization over Parallel Queues : Channel - Blind Scheduling , Restless Bandit , and Optimal Delay

This dissertation addresses several optimal stochastic scheduling problems that arise in partially observable wireless networks and multi-class queueing systems. They are single-hop network control problems under different channel connectivity assumptions and different scheduling constraints. Our goals are two-fold: To identify stochastic scheduling problems of practical interest, and to develop analytical tools that lead to efficient control algorithms with provably optimal performance. In wireless networks, we study three sets of problems. First, we explore how the energy and timing overhead due to channel probing affects network performance. We develop a dynamic channel probing algorithm that is both throughput and energy optimal. The second problem is how to exploit time correlations of wireless channels to improve network throughput when instantaneous channel states are unavailable. Specifically, we study the network capacity region over a set of Markov ON/OFF channels with unknown current states. Recognizing that this is a difficult restless multi-armed bandit problem, we construct a non-trivial inner bound on the network capacity region by randomizing well-designed round robin policies. This inner bound is considered as an operational network capacity region because it is large and easily achievable. A queuedependent round robin policy is constructed to support any throughput vector within the inner bound. In the third problem, we study throughput utility maximization over partially observable Markov ON/OFF channels (specifically, over the inner bound provided in the previous problem). It has applications in wireless networks with limited channel probing capability, cognitive radio networks, target tracking of unmanned aerial